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Line art on a banknote or other security document is slightly 
changed to effect the encoding of plural-bit digital data (i.e. 
the banknote is digitally watermarked). When such a ban- 
knote is thereafter scanned, the resulting image data can be 
recognized as corresponding to a banknote by detection of 
the encoded data. (Alternatively, the image data can be 
recognized as corresponding to a banknote by machine 
detection of other forms of watermarking, or by reference to 
visible structures characteristic of banknotes.) In response to 
detection of a banknote, the detecting apparatus (e.g., a 
scanner, photocopier, or printer) can intervene so as to 
prevent or deter counterfeiting. For example, the apparatus 
can interrupt a copying operation, and display a message 
reminding the operator that it is illegal to reproduce cur- 
rency. Another option is to dial a remote service and report 
the attempted reproduction of a banknote. Yet another option 
is to permit the copying, but to insert forensic tracer data 
(e.g. steganographically encoded binary data) in the result- 
ant copy. The tracer data can memorialize the serial number 
of the machine that made the copy and/or the date and time 
the copy was made. To address privacy concerns, such tracer 
data is not normally inserted in photocopied output, but is so 
inserted only when the subject being photocopied is detected 
as being a banknote. 

23 Claims, 3 Drawing Sheets 
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METHODS AND DEVICES FOR 
RECOGNIZING BANKNOTES AND 
RESPONDING ACCORDINGLY 

RELATED APPLICATION DATA 

This application is a divisional application of utility 
application Ser. No. 09/074,034, filed May 6, 1998, which is 
a continuation-in-part of application Ser. No. 08/438,159, 
filed May 8, 1995 (now U.S. Pat. No. 5,850,481), and now 
abandoned provisional application 60/082,228, filed Apr. 16, 
1998. 

This application is also a continuation-in-part of applica- 
tion Ser. No. 09/234,780, filed Jan. 20, 1999, now 
abandoned, which claims priority to provisional application 
No. 60/071,983, filed Jan. 20, 1998. 

The specification of application Ser. No. 08/438,159 is 
substantially identical to that of applicant's issued U.S. Pat. 
Nos. 5,636,292, 5,710,834, 5,748,763, 5,748,783. The speci- 
fication of application Ser. No. 06/082,228 is attached hereto 
as Appendix A.) 

A sibling divisional application, Ser. No. 09/293,602 
(now U.S. Pat. No. 6,285,776), is filed herewith and is 
entitled Methods for Identifying Equipment Used in Coun- 
terfeiting. 

The subject matter of this application is also related to that 
of the present assignee's other pending applications, namely 
Ser. No. 08/508,083 (now U.S. Pat. No. 5,841,978), Ser. No. 
08/534,005 (now U.S. Pat. No. 5,832,119), Ser. No. 08/637, 
531 (now U.S. Pal. No. 5,822,436), Ser, No. 08/649,419 
(now U.S. Pat. No. 5,862,260), Ser. No. 08/746,613 (now 
U.S. Pat. No. 6,122,403), Ser. No. 08/951,858 (now U.S. 
Pat. No. 6,026,193), and Ser. No. 08/969,072 (now U.S. Pat. 
No. 5,809,160), and die allowed applications referenced 
below application Ser. No. 08/327,426 (now U.S. Pat. No. 
5,768,426), Ser. No. 08/508,083 (now U.S. Pat. No. 5,841, 
978), Ser. No. 08/438,159 (now U.S. Pat. No. 5,850,481), 
and in laid-open PCT application. 

HELD OF THE INVENTION 

The present invention relates to methods and devices for 
recognizing banknotes and responding accordingly (e.g., by 
intervening to prevent attempted banknote counterfeiting). 

BACKGROUND AND SUMMARY OF THE 
INVENTION 

Watermarking is a quickly growing field of endeavor, with 
several different approaches. The present assignee's work is 
reflected in U.S. Pat. Nos. 5,710,834, 5,636,292, 5,721,788, 
5,748,763, 5,748,783, and 5,745,604, in allowed U.S. appli- 
cations Ser. No. 08/327,426 (now U.S. Pat. No, 5,768,426), 
08/508,083 (now U.S. Pat. No. 5,841,978), Ser. No. 08/438, 
159 (now U.S. Pat. No. 5,850,481), and in laid-open PCT 
application W097/43736. Other work is illustrated by U.S. 
Pat. Nos. 5,734,752, 5,646,997, 5,659,726, 5,664,018, 
5,671,277, 5,687,191, 5,687,236, 5,689,587, 5,568,570, 
5,572,247, 5,574,962, 5,579,124, 5,581,500, 5,613,004, 
5,629,770, 5,461,426, 5,743,631, 5,488,664, 5,530,759, 
5,539.735, 4,943,973, 5,337,361, 5,404,160, 5,404,377, 
5,315,098, 5,319,735, 5,337,362, 4,972,471, 5,161,210, 
5,243,423, 5,091,966, 5,113,437, 4,939,515, 5,374,976, 
4,855,827, 4,876,617, 4,939,515, 4,963,998, 4,969,041, and 
published foreign applications WO 98/02864, EP 822,550, 
WO 97/39410, WO 96/36163, GB 2,196,167, EP 777,197, 
EP 736,860, EP 705,025, EP 766,468, EP 782,322, WO 
95/20291, WO 96/26494, WO 96/36935, WO 96/42151, 
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wo 97/22206, WO 97/26733, Some of the foregoing pat- 
ents relate to visible watermarking techniques. Other visible 
watermarking techniques (e.g. data glyphs) are described in 
U.S. Pat. Nos. 5,706,364, 5,689,620, 5,684,885, 5,680,223, 
5 5,668,636, 5,640,647, 5,594,809. 

Most of the work in watermarking, however, is not in the 
patent literature but rather in published research. In addition 
to the patentees of the foregoing patents, some of the odier 
workers in this field (whose watermark-related writings can 
10 by found by an author search in the INSPEC database) 
include I. Pitas, Eckhard Koch, Jian Zhao, Norishigc 
Morimoto, Laurence Boney, Kineo Matsui, A. Z, Ttrkel, 
Fred Mintzer, B. Macq, Ahmed H. Tewfik, Frederic Jordan, 
Naohisa Komatsu, and Lawrence O' Gorman. 

The artisan is assumed to be famUiar with the foregoing 
prior art. 

In the following disclosure it should be understood that 
references to watermarking encompass not only the assign- 
ee's watermarking technology, but can likewise be practiced 

^ with any other watermarking technology, such as those 
indicated above. 

Watermarking can be applied to myriad forms of infor- 
mation. The present disclosure focuses on its applications to 
line art imagery, of the sort typically employed in banknotes, 
travelers checks, passports, stock certificates, and the hke 
(hereafter collectively referred to as "banknotes"). However, 
it should be recognized that the principles discussed below 
can also be applied outside this particular field, 

3Q Most of the prior art in image watermarking has focused 
on pixelated imagery (e.g. bit-mapped images, JPEG/MPEG 
imagery, VGA/SVGA display devices, etc.). In most water- 
marking techniques, the luminance or color values of com- 
ponent pixels are slightly changed to effect subliminal 

35 encoding of binary data through the image. (This encoding 
can be done directly in the pixel domain, or in another 
domain, such as the OCT domain.) In some systems, isolated 
pixels are changed in accordance with one or more bits of 
the binary data; in others, plural domain-related groupings 
of pixels (e.g. locally adjoining, or corresponding to a given 
DCT component) are so changed. In all cases, however, 
pixels have served as the ultimate carriers of the embedded 
data. 

While pixelated imagery is a relatively recent 

45 development, line art goes back centuries. One famUiar 
example is U.S. paper currency. On the one dollar banknote, 
for example, line art is used in several different ways. One 
is to form intricate webbing patterns around the margin of 
the note (generally comprised of light lines on dark 

50 background). Another is so form grayscale imagery, such as 
the portrait of George Washington (generally comprised of 
dark lines on a light background). 

In an illustrative embodiment of the present invention, the 
line art on a banknote or other security document is slightly 

55 changed to effect the encoding of phiral-bit digital data (i.e. 
the banknote is digitally watermarked). When such a ban- 
knote is thereafter scanned, the resulting image data can be 
recognized as corresponding to a banknote by detection of 
the encoded data. (Alternatively, the image data can be 

60 recognized as corresponding to a banknote by machine 
detection of other forms of watermarking, or by reference to 
visible structures characteristic of banknotes.) 

In response to detection of a banknote, the detecting 
apparatus (e.g., a scanner, photocopier, or printer) can inter- 

65 vene so as to prevent or deter counterfeiting. For example, 
the apparatus can intermpt a copying operation, and display 
a message reminding the operator that it is illegal to repro- 
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duce currency. Another option is to dial a remote service and the luminosity of regions 12 as the algorithm would other- 
report the attempted reproduction of a banknote. Yet another wise change tiie luminance or colors of pixels in a pixelated 
option is to pemait the copying, but to insert forensic tracer image. 

data (e.g. steganographically encoded binary data) in the in an exemplary algorithm, the binary data is represented 

resultant copy. The tracer data can memorialize the serial 5 as a sequence of -Is and Is, instead of Os and Is. (The binary 

number of the machine that made the copy and/or the date data can comprise a single datum, but more typically com- 

and time the copy was made. To address privacy concerns, prises several. In an illustrative embodiment, the data com- 

such tracer data is not normally inserted in photocopied prises 100 bits.) 

output, but is so inserted only when the subject being g^ch element of the binary data sequence is then multi- 
photocopied is detected as being a banknote. lo ^^^^ g corresponding element of a pseudo-random 

The foregoing features and advantages of the invention number sequence, comprised of -Is and Is, to yield an 

will be more readily apparent from the following detailed intermediate data signal. Each element of this intermediate 

description, which proceeds with reference to the accompa- data signal is mapped to a corresponding sub -part of the 

nying drawings. image, such as a region 12. The image in (and optionally 

around) this region is analyzed to determine its relative 

BRIEF DESCRIPTION OF THE DRAWINGS capabiUty to conceal embedded data, and a corresponding 

HGS. lAand IB show prior art techniques for achieving scale factor is produced. Exemplary scale factors may range 

, ^ . 1 . ^ . ^ ^ from 0 to 3. The scale factor for the region is then multiplied 

grayscale effects using Ime art. by the element of the intermediate data signal mapped to the 

no. 2 shows a virtual anay of grid points that can be 20 region in order to yield a "tweak" value for the region. In the 

imposed on an image accordmg to one embodiment of the in^si^ated case, the resulting tweaks can range from -3 to 3. 

present invention. luminosity of the region is then adjusted in accordance 

FIG. 3 shows a virtual array of regions that can be the tweak value. A tweak value of -3 may correspond 

imposed on an image according to the FIG. 2 embodiment. a _5% change in luminosity; -2 may correspond lo -2% 

FIG. 4 shows an excerpt of FIG. 3 with a line from a line change; -1 may correspond to -1% change; 0 may corrc- 

art image passing therethrough. spond to no change; 1 may correspond to +1% change; 2 

FIG. 5 shows changes to the width of the line of FIG. 3 may correspond to +2% change, and 3 may correspond to 

to effect watermark encoding according to one embodiment +5% change. (This example follows the basic techniques 

of the present invention. described in the Real Time Encoder embodiment disclosed 

FIG. 6 shows changes to the position of the hue of FIG. ^ 5,710,834.) . _ , 

3 to effect watermark encoding according to another In FIG. 5, the watermarking algorithm determined that the 

embodiment of the present invention. luminance of region A should be reduced by a certain 

n A * percentage, while the lummance of regions C and D should 

no. 7 IS a block diag^m of a photocopier according to ^^^^6^ percentages. 

another embodiment of the mvention. 35 , • * i • ■ j j l ■ - *u 

In region A, the luminance is reduced by increasing the 

DETAILED DESCRIPTION line width. In region D, the luminance is increased by 

reducing the Une width; similarly in region C (but to a lesser 
Referring to FIG. 2, an illustrative form of the invention extent) 
employs a grid 10 of unaginary reference points arrayed j.^^ ^ .^^ ^ ^^^^ ^ 
over a Ime art image. TTie spacing between pomts is 250 im ^ luminance. Hiis is not fatal to the 
m the Illustrated arrangement, but greater or lesser spacings J^^^^ ^^^^^^^^ ^^^^ watermarking algorithm redun- 
can of course be used. ^^^^ encodes each bit of data in sub-parts spaced through- 
Associated with each grid point is a surroimding region ^ image 
12, shown in FIG. 3. As described below, the luminosity (or ^.^^^ ^ ^^^.^^^ ^ ^ 5 
reflectance) of each of these regions 12 is slightly changed exaggerated for purposes of illustration. While the 
to effect the subHminal encodmg of binary data. illustrated variance is possible, most implementations will 
Region 12 can take various shapes; the illustrated modulate the line width 3-50% (increase or decrease), 
rounded-rectangular shape is representative only. (The ilius- (Many watermarking algorithms routinely operate within a 
trated shape has the advantage of encompassing a fairly ^-g^^^ margin of about +/-!% changes in luminosity to 
large area while introducing fewer visual artifacts than, e.g., ^gf^^ encoding. That is, the "noise" added by the encoding 
square regions.) In other embodiments, squares, rectangles, amounts to just 1% or so of the underlying signal. Lines 
circles, ellipses, etc., can alternatively be employed. typically don't occupy the full area of a region, so a 10% 
FIG. 4 is a magnified view of an excerpt of FIG. 3, change to line width may only effect a 1% change to region 
showing a line 14 passing through the grid of points. The 55 luminosity, etc. Banknotes are different from photographs in 
width of the line, of course, depends on the particular image that the art need not convey photorealism. Tlius, banknotes 
of which it is a part. The illustrated line is about 25 im in can be encoded with higher energy than is used in water- 
width; greater or lesser widths can naturally be used. marking photographs, provided the result is still aestheti- 
In a first embodiment of the invention, shown in FIG. 5, cally satisfactory. To illustrate, localized luminance changes 
the width of the line is controllably varied so as to change go on the order of 10% are possible in banknotes, while such a 
the luminosity of the regions through which it passes. To level of watermark energy in photographs would generally 
increase the luminosity (or reflectance), the line is made be considered unacceptable. In some contexts, localized 
narrower (i.e. less ink in the region). To decrease the luminance changes of 20, 30, 50 or even 100% are 
luminosity, the line is made wider (i.e. more ink). acceptable.) 

Whether the luminance in a given region should be 65 In the illustrated embodiment, the change to line width is 

increased or decreased depends on the particular watermark- a function solely of the tweak to be applied to a single 

ing algorithm used. Any algorithm can be used, by changing region. Thus, if a Une passes through any part of a region to 
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which a tweak of 2% is lo be applied, the line width in that 
region is changed to effect the 2% luminance difference. In 
variant embodiments, the change in line width is a function 
of the line's position in the region. In particular, the change 
in line width is a function of the distance between the 
region's center grid point and the line's closest approach to 
that point. If the line passes through the grid point, the full 
2% change is effected. At successively greater distances, 
successively less change is applied. The manner in which the 
magnitude of the tweak changes as a function of line 
position within the region can be determined by applying 
one of various interpolation algorithms, such as the bi-Unear, 
bi-cubic, cubic splines, custom curve, etc. 

In other variant embodiments, the change in line width in 
a given region is a weighted function of the tweaks for 
adjoining or surrounding regions. Thus, the line width in one 
region may be increased or decreased in accordance with a 
tweak value corresponding to one or more adjoining regions. 

Combinations of the foregoing embodiments can also be 
employed. 

In the foregoing embodiments, it is sometimes necessary 
to trade-off the tweak values of adjoining regions. For 
example, a line may pass along a border between regions, or 
pass through the point equidistant from four grid points 
("equidistant zones"). In such cases, the line may be subject 
to conflicting tweak values — one region may want to 
increase the line width, while another may want to decrease 
the line width. (Or both may want to increase the line width, 
but differing amounts.) Similarly in cases where the line 
does not pass through an equidistant zone, but the change in 
line width is a function of a neighborhood of regions whose 
tweaks are of different values. Again, known interpolation 
functions can be employed to determine the weight to be 
given the tweak from each region in determining what 
change is to be made to the line width in any given region. 

In the exemplary watermarking algorithm, the average 
change in luminosity across the image is zero, so no gen- 
eralized lightening or darkening of the image is apparent. 
The locaHzed changes in luminosity are so minute in 
magnitude, and localized in position, that they are essen- 
tially invisible (e.g. inconspicuous/subliminal) to human 
viewers. 

An alternative embodiment is shown in FIG. 6, in which 
line position is changed rather than hne width. 

In FIG. 6 the original position of the line is shown in 
dashed form, and the changed position of the line is shown 
in solid form. To decrease a region's luminosity, the line is 
moved slightly closer to the center of the grid point; to 
increase a region's luminosity, the Hne is moved slightly 
away. Thus, in region A, the hne is moved towards the center 
grid point, while in region D it is moved away. 

It will be noted that the hne on the left edge of region A 
does not return to its nominal (dashed) position as it exits the 
region. This is because the region to the left of region A also 
is to have decreased luminosity. Where possible, it is gen- 
erally preferable not to return a line to its nominal position, 
but instead permit shifted lines to remain shifted as they 
enter adjoining regions. So doing permits a greater net line 
movement within a region, increasing the embedded signal 
level. 

Again, the line shifts in FIG. 6 are somewhat exaggerated. 
More typical hne shifts are on the order of 3-50 im. 

One way to think of the FIG. 6 embodiment is to employ 
a magnetism analogy. The grid point in the center of each 
region can be thought of as a magnet. It either attracts or 
repels lines. A tweak value of -3, for example, may corre- 
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spond to a strong-valued attraction force; a tweak value of 
+2 may correspond to a middle -valued repulsion force, etc. 
In FIG. 6, the grid point in region A exhibits an attraction 
force (i.e. a negative tweak value), and the grid point in 
5 region D exhibits a repulsion force (e.g. a positive tweak 
value). 

The magnetic analogy is useful because the magnetic 
effea exerted on a line depends on the distance between the 
Unc and the grid point. Thus, a Hne passing near a grid point 
10 is shifted more in position than a Line near the periphery of 
the region. 

Each of the variants discussed above in connection with 
FIG. 5 is likewise applicable to FIG. 6. 

Combinations of the embodiments of FIGS. 5 and 6 can 
of course be used, resulting in increased watermark energy, 
better signal-to-noise ratio and, in many cases, less notice- 
able changes. 

In still a further embodiment, the luminance in each 
region is changed while leaving the hne unchanged. This can 
be effected by sprinkling tiny dots of ink in the otherwise - 
vacant parts of the region. In high quality printing, of the 
type used with banknotes, droplets on the order of 3im in 
diameter can be deposited. (StiU larger droplets are still 
beyond the perception threshold for most viewers.) Speck- 
ling a region with such droplets (either in a regular array, or 
random, or according to a desired profile such as Gaussian), 
can readily effect a 1% or so change in luminosity. (Usually 
dark droplets are added to a region, effecting a decrease in 
luminosity. Increases in luminosity can be effected by speck- 
ling with a light colored ink, or by forming light voids in Une 
art otherwise present in a region.) 

In a variant of the speckling technique, very thin mesh 
lines can be inserted in the artwork — again to slightly 
change the luminance of one or more regions. 

Although not previously mentioned, it is contemplated 
that the banknote wiU include some manner of calibration 
information to facilitate registration of the image for decod- 
ing. This calibration information can be steganographic or 
4Q overt. Several techniques for sleganographically embedding 
calibration information are disclosed in my prior patents and 
apphcations. Other techniques can be found in others of the 
cited work. 

To decode watermark data, the encoded line art image 
45 must be converted into electronic form for analysis. This 
conversion is typically performed by a scanner. 

Scanners are well known, so a detailed description is not 
provided here. Suffice it to say that scanners conventionally 
employ a line of closely spaced photodetector cells that 
50 produce signals related to the amount of the light reflected 
from successive swaths of the image. Most inexpensive 
consumer scanners have a resolution of 300 dots per inch 
(dpi), or a center to center spacing of component photode- 
tectors of about 84 im. Higher quality scanners of the sort 
55 found in most professional imaging equipment and photo- 
copiers have resolutions of 600 dpi (42 im), 1200 dpi (21 
im), or better. 

Taking the example of a 300 dpi scanner (84 im photo - 
detector spa cing), each 250 im region 12 on the banknote 

60 will correspond to about a 3x3 array of photodetector 
samples. Naturally, only in rare instances will a given region 
be physicafly registered with the scanner so that nine pho- 
todetector samples capture the luminance in that region, and 
nothing else. More commonly, the line art is skewed with 

65 respect to the scanner photodetectors, or is longitudinally 
misahgned (i.e. some photodetectors image sub-parts of two 
adjoining regions). However, since the scanner oversamples 
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the regions, the himinance of each region can unambigu- Another hardware device that can employ the foregoing 

ously be determined. principles is a standalone scanner. A programmed processor 

In one embodiment, the scanned data from the line art is (or dedicated hardware) inside the scanner analyzes the data 

collected in a two dimensional array and processed— being generated by the device, and responds accordingly, 
according to one of the techniques disclosed in my prior ^ Yet another hardware device that can employ the forego- 

patents and applications — to detect the embedded calibra- ing principles is a printer. A processor inside the device 

tion information. The array is then processed to effect a analyzes graphical image data to be printed, looking for 

virtual re-registration of the image data. A software program watermarks associated with banknotes, 

then analyzes the statistics of the re-registered data (using por both the scanner and printer devices, response strat- 
the techniques disclosed in my prior writings) to extract the lO egies can include disabling operation, or inserting tracer 

bits of the embedded data. information. (Such devices typically do not have dial-out 

(Again, the reference to my earlier watermark decoding capabilities.) 

techniques is exemplary only. Once scanning begins and the Again, it is desirable to process the scanner or prmter data 

data is available in pixel form, it is straightforward to apply as it becomes available, so as to detect any banknote 

any other watermark decoding technique to extract a processing with a minimum of delay. Again, there will be 

correspondingly-encoded watermark.) some lag time before a detection decision is made. 

In a variant embodiment, the scanned data is not Accordingly, the scanner or printer output wUl be comprised 

assembled in a complete array prior to the processing. of two parts, one without the tracer data, and another with 

Instead, it is processed in real -lime, as it is generated, in the tracer data. 

order to detect embedded watermark data without delay. Banknotes presently include various visible structures 

(Depending on the parameters of the scanner, it may be xh?ii can be used as aids in banknote detection (e.g. the seal 

necessary to scan a half-inch or so of the line art image of the issuing central bank, and various geometrical 

before the statistics of the resulting data unambiguously markings). In accordance with a further aspect of the present 

indicate the presence of a watermark.) invention, a banknote is analyzed by an integrated system 

In accordance with another aspect of the invention, van- that considers both the visible structures and watermark- 

ous hardware devices are provided with the capability to embedded data. 

recognize embedded watermark data in any hne art images Visible banknote structures can be sensed using known 

they process, and to respond accordingly. pattern recognition techniques. Examples of such techniques 

One example is a color photocopier. Such devices employ 30 are disclosed in U.S. Pat. Nos. 5,321,773, 5,390,259, 5,533, 

a color scanner to generate sampled (pixel) data correspond- 144, 5,539,841, 5,583,614, 5,633,952, 4,723,149 and 5,424, 

ing to an input media (e.g. a dollar bill). If watermark data 807 and laid-open foreign application EP 766,449. 

associated with a banknote is detected, the photocopier can Iq photocopiers (and the like) equipped to detect both 

take one or more steps. visible and watermarked banknote markings, the detection 

One option is simply to interrupt copying, and display a of either causes one or more of the above -noted banknote 

message reminding the operator that it is illegal to reproduce responses to be initiated. 

currency. Again, scanners and printers can be equipped with a 

Another option is to dial a remote service and report the similar capability-analyzing the data for either of these 

attempted reproduction of a banknote. Photocopiers with banknote hallmarks. If either is detected, the software (or 

dial-out capabiUties are known in the art (e.g. U.S. Pat. No. hardware) responds accordingly. 

5,305,199) and are readily adapted to this purpose. The Identificationof banknotes by watermark data provides an 

remote service can be an independent service, or can be a important advantage over recognition by visible 

government agency. structures — it cannot so easily be defeated, A banknote can 

Yet another option is to pennit the copying, but to insert be doctored (e.g. by white-out, scissors, or less crude 
forensic tracer data in the resultant copy. This tracer data can techniques) to remove/obliterate the visible structures. Such 
take various forms. Steganographically encoded binary data a document can then be freely copied on cither a visible 
is one example. An example is shown in U.S. PAT. No, structure -sensing photocopier or scanner/printer installation. 
5,568,268. The tracer data can memorialize the serial num- The removed visible structure can then be added in via a 
ber of the machine that made the copy and/or the date and second printing/photocopying operation. If the printer is not 
time the copy was made. To address privacy concerns, such equipped with banknote-disabling capabilities, image- 
tracer data is not normally inserted in photocopied output, editing tools can be used to insert visible structures back into 
but is so inserted only when the subject being photocopied image data sets scanned from such doctored bills, and the 
is detected as being a banknote. (Such an arrangement is complete bill freely printed. By additionally including 
shown in FIG. 7.) embedded watermark data in the banknote, and sensing 

Desirably, the scan data is analyzed on a line-by-line basis same, such ruses will not succeed, 
in order to identify illicit photocopying with a minimum of (A similar ruse is to scan a banknote image on a non- 
delay. If a banknote is scanned, one or more lines of scanner banknote-sensing scanner. The resulting image set can then 
output data may be provided to the photocopier's repro- be edited by conventional image editing tools to remove/ 
graphic unit before the banknote detection decision has been obliterate the visible structures. Such a data set can then be 
made. In this case the photocopy will have two regions: a printed — even on a printer/photocopier that examines such 
first region that is not tracer-marked, and a second, subse- data for the presence of visible structures. Again, the missing 
quent region in which the tracer data has been inserted. visible structures can be inserted by a subsequent printing/ 

Photocopiers with other means to detect not-to-be-copied photocopying operation.) 
documents are known in the art, and employ various 65 Desirably, the visible structure detector and the water- 
response strategies. Examples are detailed in U.S. Pat. Nos. mark detector are integrated together as a single hardware 
5,583,614, 4,723,149, 5,633,952, 5,640,467, and 5,424,807. and/or software tool. This arrangement provides various 
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economies, e.g., in iaterfacing with the scaoner, maDipulat- 
ing pixel data sets for pattern recognitioQ and watermark 
extraction, electroQically re-registering the image to facili- 
tate pattern recognition/watermark extraction, issuing con- 
trol signals (e.g. disabling) signals to the photocopier/ 5 
scanner, etc. 

(To provide a comprehensive disclosure without uaduly 
leagthening the following specification, applicants incorpo- 
rate by reference the patent documents cited above (except 
applications Ser. Nos. 09/234,780 and 60/071,983).) 

From the foregoing, it will be recognized that embodi- 
ments according to the present invention allow banknotes to 
be recognized based on optical scan data, and responsive 
actions taken, thereby reducing the risk of counterfeiting. 
Other devices, responsive to authentic banknotes, are 
enabled. 

Having described and illustrated the principles of my 
invention with reference to several illustrative embodiments, 
it will be recognized that these embodiments are exemplary ^ 
only and shotild not be taken as limiting the scope of my 
invention. Guided by the foregoing teachings, it should be 
apparent that other watermnarking, decoding, and anti- 
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counterfeiting technologies can be substituted for, and/or 
combined with, the elements detailed above to yield similar 
effects. 

While the invention has been described with reference to 
embodiments employing regular rectangular arrays of grid 
points, those skilled in the art will recognize that other arrays 
of points — neither rectangular nor regular — can alterna- 
tively be used. 

While the invention has been described with reference to 
embodiments that scale the embedded energy in accordance 
with local image characteristics, in other embodiments a 
manually crafted energy profile can be implemented. That is, 
a mask defining embedded signal magnitudes at different 
parts of the image can be manually devised, and employed 
to tailor the change in luminance in each region. 

In view of the many possible embodiments to which the 
principles of the invention may be put, it should be recog- 
nized that the detailed embodiments are illustrative only and 
should not be taken as Hmiting the scope of my invention. 
Rather, I claim as my invention all such embodiments as 
may come within the scope and spirit of the following claims 
and equivalents thereto. 



APPENDIX A 



WATERMARKING METHODS, APPARATUSES, AND APPLICATIONS 

(To provide a comprehensive disclosure withaut uaduJy leagthening the following specification, 
applicants incorporate by reference the cited patent documents.) 

Watermarking is a quickly growing field of endeavor, with several different approaches. The 
present assignee's work is reflected in U.S. Pat Nos. 5,730,834, 5,636,292, 5,721,788, allowed U.S. 
applications 08/327,426, 08/598,083, 08/436,134 (to issue as patent 5,748,763), 08/436,102 (to issue as 
patent 5,748,783), and 08/634,521 (to issue as patent 5,745,604), and laid open PCT application 
WO 97/43736. Other work is illustrated by U.S. Pat. Nos. 5,734,752, 5,646,997, 5,659,726, 5,664,018, 
5,673,277, 5,687,391, 5,687,236, 5,689,587, 5,568,570, 5,572,247, 5,574,962, 5,579,124, 5,581^00, 
5,613,004, 5,629,770, 5,463,426, 5,743,631, 5,488,664, 5,530,759, 5,539,735, 4,943,973, 5,337,361, 
5,404,360, 5,404,377, 5,335,098, 5,319,735, 5,337,362, 4,972,471, 5,161,210, 5,243,423, 5,091,966, 
. (0-5,113,437, 4,939,515, 5,374,976, 4,855,827, 4,876,617, 4,939,515, 4,963,998, 4,969,041, and published 
foreign applicaUons WO 98/02864, EP 822,550, WO 97/39410, WO 96/361, GB 2,196,167, EP 
777,197, EP 736,860, EP 705.025, EP 766,468, EP 782,322, WO 95/20291, WO 96/26494, WO 96/36935, 
WO 96/42153, WO 97/22206, WO 97/26733. Some of the foregoing patents relate to visible watermarking 
techniques. Other visible watcrmaiking techniques (e.g. data glyphs) arc described in U.S. PaL 
Nos. 5,706,364, 5,689.620. 5.684,885. 5.680,223, 5,668,636, 5.640,647, 5,594,809. 
Most of the work in watermarking, however, is not in the patent literature but rather in published 
research. In addition to the patentees of the foregoing patents, some of the other workers in this field 
(whose watermark-related writings can by found by an author search in the INSPEC database) include I. 
Pitas, Eckhaid Koch, Jian Zhao, Norishige Morimoto, Laurence Boney, Kinco Matsui, A. Z. Hrkcl, Fred 
z^- Mint2eT,-B. Macq, Ahmed H. Tewfik, Frederic Jordan, Naohisa Komatsu, and Lawrence O'Gorman. 
The artisan is assumed to be familiar with the foregoing prior art. 

In the following disclosure it should be understood that references to watermarking encompass 
not only the assignee's watermarking technology, but can likewise be practiced with any other 
watermarking technology, such as those indicated above. 
''Watermarking can be applied to myriad forms of information. These include imagery (including 
^ video) and audio - whether represented in digital form (e.g. an image comprised of pixels, digital video, 
etc.), or in an analog representation (e.g. nonsampled music, printed imagery, banknotes, etc.) 
Watermarking can be applied to digital content (e.g, imagery, audio) either before or after compression. 
Watermarking can also be used in various "description" or "synthesis" language representations of 

'Lo content, such as Structured Audio, Csound, NetSound, SNHC Audio and the like (c,f. 

http://sound-media.mit.edu/mpeg4/) by specifying synthesis commands that generate watermark data as 
well as the intended audio signal. Watennarldng can also be applied to ordinary media, whether or not it 
conveys information. Examples include paper, plastics, laminates, paper/film emulsions, etc. A 
watermark can embed a single bit of information, or any number of bits. 

^Cj" The physical manifestation of watermarked information most commonly takes the form of altered 

7 signal values, such as sUghtly changed pixel values, picture luminance, picture colors, DCT coefficients, 
instantaneous audio amplitudes, etc. However, a watermark can also be manifested in other ways, such 
as changes in the surface microtopology of a medium, localized chemical changes (e.g. in photographic 
emulsions), localized variations in optical density, localized changes in luminescence, etc. ^^^teTmarks 

. ^ can also he opticaUy implemented in holograms and conventional paper watermarks. 

^ One improvement to existing technology is to employ established web crawler services {e.g. 

Alta \^ta, Excite, or Inktomi) to search for watermarked content (on the Web, in internet news groups, 
BBS systems, on-lme systems, etc.) in addition to their usual data collecting/indexing operations. Such 
crawlers can download files that may have embedded watermarks (c.g. *.JPG, *.WAV, etc.) for later 
analysis. These files can be processed, as described below, in real time. More commonly, such files are 
queued and processed by a computer distinct from the crawler computer. Instead of performing 
watermark-read operations on each such file, a screening technique can be employed to identify those 
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WATEEIMARKING METHODS, APPARATUSES, AND APPUCAnONS 



most liiccly to be conveying watcnnark data. One such technique is to perform a DCT operation on an 
image, and look for spectral coefficients associated with certain watermarking techniques (e,g. 
coefiBcients associated with an inclined embedded subliminal grid). Tb decode spread spectrum based 
watermarks, the analyzing computer requires access to the noise signal used to spread the data signal. In 
one embodiment, interested parties submit their noise/key signals to the crawler service so as to enable 
their marked content to be located. The crawler service maintains such information in confidence, and 
uses different noise signals in decoding an image (image is used herein as a convenient shorthand for 
imagery, video, and audio) until watermarked data is found (if present). This allows the use of web 
aawlers to locate content with privately-coded watermarks, instead of just publicly-coded watermarks as 
is presently the case. The queueing of content data for analysis provides certain opportunities for 
computational shortcuts. For example, like sized images (e.g. 256 x 256 pixels) can be tiled into a larger 
image, and examined as a unit for the presence of watermark data. If the decoding technique (or the 
optional prescreening technique) employs a DCT transform or the like, the block size of the transfonn 
can be tailored to correspond to the tile size (or some integral fraction thereoQ. Blocks indicated as 
lUcely having watermarls can then be subjected to a full read operation. If the queued data is sorted by 
file name, file size, or checksum, duplicate files can be identifi.ed. Once such duplicates are identified, the 
analysis computer need consider only one instance of the file. If watermark data is decoded from such a 
file, the content provider can be informed of each URL at which copies of the file were found 
Some commentators have observed that web crawler-based searches for watermarked images 
can be defeated by breaking a watermarked image into sub-blocks (tiles). HTML instructions, or the like, 
cause the sub-blocks to be presented in tiled fashion, recreating the complete image. However, due to the 
small size of the component sub-blocks, watermark reading is not reliably accomplished. 
This attack is overcome by instructing the web crawler to collect the display instructions (e.g. 
HTML) by which image files are positioned for display on a web page, in addition to the image 
themselves. Before files collected from a web page are scrutinized for watermarks, they can be 
concatenated in the arrangement specified by the display instructions. By this arrangement, the tiles are 
reassembled, and the watermark data can be reliably recovered. 

Another such postulated attack against web crawler detection of image watermarks is to 
scramble the image (and thus the watermark) in a file, and employ a Java applet or the like to unscramble 
the image prior to viewing. Existing web crawlers inspect the file as they find it, so the watermark is not 
detected. However, just as the Java descrambling applet can be invoked when a user wishes access to a 
file, the same applet can similarly be employed in a web crawler to overcome such attempted 
circumvention of watermark detection. 

Althougji "content" can be located and indexed by various web crawlers, the contents of the 
"content" are unknown. A * JPG file, for example, may include pornography, a photo of a sunset, etc 
Watermarks can be used to indelibly associate meta-data within content (as opposed to stored in 
a data structure that forms another part of the object, as is conventionally done wiUi meta-data). The 
watermark can include text saying "sunsetf or the like. More compaa information representations can 
alternatively be employed (e.g. coded references). Still further, the watermark can include (or consist 
entirely of) a Unique ID (UID) that serves as an index (key) into a network connected remote database 
containing the mcta data descriptors. By such arrangements, web crawlers and the like can extract and 
index the meta-data descriptor tags, allowing searches to be conducted based on semantic descriptions of 
the file contents, rather than just by file name. 

Existing watermarks commonly embed information serving to communicate copyright 

information. Some systems embed text identifying the copyright holder. Others embed a UID which is 

used as an index into a database where the name of the copyright owner, and associated information, is 

stored. 

Looking ahead, watermarks should serve more than as silent copyright notices. One option is to 
use watennarks to embed *'intelligence" in content One form of intelligence is knowing its *'home." 
"Home" can be the URL of a site with which the content is associated. A photograph of a car, for 
example, can be watermarked with data identifying the web site of an auto-dealer that published the 
image. Wherever the image goes, it serves as a link back to the original disseminator. The same 
technique can be applied to corporate logos. Wherever they are copied on the internet, a suitably 
equipped browser or the like can decode the data and link back to the corporation's home page. 
(Decoding may be effected by positioning the cursor over the logo and pressing the right- mouse button, 
which opens a window of options-one of which is Decode Watermark.) 
To reduce the data load of the watermark, the intelligence need not be wholly encoded in the 
content's watermark. Instead, the watermark can again provide a UID - this time identifying a remote 
database record where the URL of the car dealer, etc., can be retrieved. In this manner, images and the 
like become marketing agents - linking consumers with vendors (with some visvml salesmanship thrown 
in). In contrast to the copyright paradigm, in which dissemination of imagery was an evil sought to be 
tracked and stopped, dissemination of the imagery can now be treated as a selling opportunity. A 
watermarked image becomes a portal to a commercial transaction. 

(Using an intermediate database between a watermarked content file and its ultimate home (le. 
indirect linking) serves an important advantage: it allows the disseminator to change the "home" simply 
by updating a record in the database. Thus, for example, if one company is acquired by another, the 
former company's smart images can be made to point to the new company's home web page by updating 
a database record. In contrast, if the old company's home URL is hard-coded (i.e. watennarked) in the 
object, it may point to a URL that eventually is abandoned. In this sense, the intermediate database 
serves as a switchboard that couples the file to its current home. 

The foregoing techniques arc not limited to digital content files. The same approach is equally 
applicable with printed imagery, etc. A printed catalog, for example, can include a picture illustrating a 
jacket. Embedded in the picture is watermarked data. This data can be extracted by a simple hand 
scanner/decoder device using straightforward scanning and decoding techniques (e.g. those known to 
artisans in those fields). In watermark- reading applications employing hand-scanners and the like, it is 
important that the watermark decoder be robust to rotation of the image, since the catalog photo will 
likely be scanned off-axis. One option is to encode subliminal graticules (e.g. visualization 



01/06/2004, EAST Version: 1.4.1 



13 



US 6,427,020 Bl 



APPENDIX A-contimied 



WATERMARKING METHODS, APPARATUSES, AND APPUCAnONS 

synchronization codes) ia the catalog photo so that the set of image data can be post processed to restore 
it to proper alignment prior to decoding. 

The scanner/decoder device can be coupled to a modem-equipped computer, a tclcphonCj or any 
other commumcations device. In the former instance, the device provides URL data to the computer's 
web browser, linking the browser to the catalog vendor's order page. CThe device need not include its 
own watermark decoder; this task can be performed by the computer.) The vendor's order page can 
detail the size and color options of the jacket, inventory availability, and solicit ordering instructions 
(credit card number, delivery options, etc.) - as is conventionally done with on-line merchants. Such a 
device connected to a telephone can dial the catalog vender's toll-free automated order-taking telephone 
number (known, e.g., from data encoded in the watermark), and identify the jacket to the order center. 
Voice prompts can then soUdt the customer's choice of size, color, and delivery options, which are input 
by Touch Tone instructions, or by voiced words (using known voice recognition software at the vendor 
facility). 

In such applications, the watermark may be conceptualized as an invisible bar code employed in 
a purchase transaction. Here, as elsewhere, the watermark can serve as a seamless interface bridging 
the print and digital worlds 

Another way of providing content with intelligence is to use the watermark to provide Java or 
Active code. The code can be embedded in the content, or can be stored remotely and linked to the 
content. When the watermarked object is activated, the code can be executed (either automatically), or at 
the option of the user). This code can peform virtually any function. One is to "phone home"- 
initiating a browser and linking to the object's home The object can then relay any manner of data to its 
home. This data can specify some attribute of the data, or its use. The code can also prevent accessing 
the underlying content until permission is received. An example is a digital movie that, when double 
clicked, automatically executes a watennarker-embedded Java applet which links through a browser to the 
movie's distributor. The user is then prompted to input a credit card number. After the number has been 
verified and a charge made, the applet releases the content of the file to the computer's viewer for 
viewing of the movie. Support for these operations is desirably provided via the computer's operating 
system, or plug-in software. 

Such arrangements can also be used to collect user-provided demographic information when 

smart image content is accessed by the consumer of the content. The demographic information can be 

written to a remote database and can be used for market research, customization of information about the 

content provided to the consumer, sales opportunities, advertising, etc. 

In audio and video and the like, watermarks can serve to convey related information, such as 

links to WWW fen sites, actor biographies, advertising for marketing (T-shirts, CDs, concert 

tickets). In such applications, it is desirable (but not necessary) to display on the user interfece (e.g. 

screen) a small logo to signal the presence of additional information. When the consumer selects the logo 

via some selection device (mouse, remote control button, etc.), the information is revealed to the 

consumer, who can then interact with it. 

Much has been written (and patenud) on the topic of asset rights management. Sample patent 
documents include U.S. Pat. Nos. 5,715.403. 5,638.443, 5,634,012, 5,629,980. Again, much of the technical 
work is memorialized in journal articles, which can be identified by searching for relevant company 
names and trademarks such as [BM's Cryptolope system, Portland Software's ZipLock system, the Rights 
Exchange service by Softbank Net Solutions, and the DigiBox system from InteiTrust Technologies. 
An exemplary asset management system makes content available (e.g. from a web server; or on a 
new computer's hard disk) in encrypted form. Associated with the encrpted content is data identifying 
the content (e.g. a preview) and data specfying various rights associated with the content. If a user wants 
to make fuller use of the content, the user provides a charge authorization (e.g. a credit card) to the 
distributor, who then provides a decryption key, allowing access to the content. (Such systems are often 
realized using object-based technology. In such systems, the content is commonly said to be distributed in 
a "secure container.") 

Desirably, the content should be marked (personalized/serialized) so that any illicit use of the 
content (after decryption) can be tracked. This marking can be performed with watermarking, which 
assures that the mark travels with the content wherever— and in whatever form - it may go. The 
watermarking can be effected by the distributor- prior to dissemination of the encrypted object- such as 
by encoding a UID that is associated in a database with that particular container. When access rights 
are granted to that container, the database record can be updated to reflect the purchaser, the purchase 
date, the rights granted, etc. An alternative is to include a watermark encoder in the software tool used to 
access (e.g. decrypt) the content. Such an encoder can embed watermark data in the content as it is 
released from the secure container, before it is provided to the user. The embedded data can include a 
UID, as described above. This UID can be assigned by the distributor prior to disseminating the 
container. Alternatively, the UID can be a data string not known or created until access rights have been 
granted. In addition to the UID, the watermark can include other data not known to the distributor, e.g. 
information specific to the times(s) and manner(s) of accessing the content. 
In other systems, access rights systems can be realized with watermarks without containers etc. 
Full resolution images for example, can be freely available on the web. If a user wishes to incorporate 
the imagery into a web page or a magazine, the user can interrogate the imagery as to its terms and 
conditions of use. This may entail linking to a web site specified by the embedded watermark (directly, or 
through an intermediate database), which specifies the desired information. The user can then arrange 
the necessary payment, and use the image knowing that the necessary rights have been secured. 
As noted, digital watermarks can also be realized using conventional (e.g. paper) watermarking 
technologies. Known techniques for watermarking media (e.g. paper, plastic, polymer) are disclosed in 
U.S. Pat. Nos. 5,536,468. 5.275,870, 4,760.239, 4.256.652, 4^70,200, and 3,985,9 and can be adapted to 
display of a visual watermark instead of a logo or the like. Note that some forms of traditional 
watermarks which are designed to be viewed with transmissive light can also show up as low level signals 
in reflective light, as is typically used in scanners. Transmissive illumination detection systems can also 
be employed to detect such watermarks, using optoelectronic traditional watermark detection 
technologies known in the art. 
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As also noted, digital watermarks can be realized as part of optical holograms. Known 
techniques for producing and securely mounting holograms are disclosed in U.S. Pat Nos. 5,319,475, 
5.694,229, 5,492,370, 5,483,363, 5,658,411 and 5,310,222. To watennark a hologram, the waUrmark 
can be represented in the image or data model from which the holographic diEfraction grating is 
produced. In oac embodiment, the hologram is produced as before, and displays an object or symbol. 
The watermark markings appear in the background of the image so that they can be detected from all 
viewing angles. In this context, it is not critical that the watermark representation be essentially 
imperceptible to the viewer. If desired, a fairly visible noise-like pattern can be tised without impairing 
the use to which the hologram is put. 

Digital watermarks can also be employed in conjunction with labels and tags. In addition to 
conventional label/tag printing processes, other techniques- tailored to security- can also be employed. 
Known techniques useful in producing security labels/lags are disclosed in U.S. Pat. Nos. 5,665,194, 
5,732,979, 5,651,615, and 4,268,983. The imperceptibility of watermarked data, and the ease of machine 
decoding, are some of the benefits associated with watermarked tags/labels. Additionally, the oist is far 
less than many related technologies (e.g. holograms). AV^tennarks in this application can be used to 
authenticate the originality of a product label, either to the merchant or to the consumer of the associated 
product, using a simple scanner device, thereby reducing the rate of counterfeit product sales. 
Recent advances in colorprinting technology have greatly increased the level of casual 
counterfeiting. High quality scanners are now readily available to many computer users, with 300 dpi 
scanners available for under $100, and 600 dpi scanners available for marginally more. Similarly, 
photographic quality color inkjet printers are commonly available from Hewlett-Packard Co., Epson, 
etc. for under $300. 

Watermarks in banknotes and other security documents passports, stock certificates, checks, 

etc. - all collectively referred to as banknotes herein) offer great promise to reduce such counterfeiting, 

as discussed more ftilly below. Additionally, watermarks provide a high-confidence technique for 

banknote authentication. One product enabled by this increased confidence is automatic teller machines 

that accept, as well as dispense, cash. The machine is provided with known optical scanning technology 

to produce digital data corresponding to the face(s) of the bill. This image set is then analyzed to extract 

the watermark data. In watermarking technologies that require knowledge of a code signal for decoding 

(e.g. noise modulation signal, cripto key, spreading signal, etc.), a bill may be watermarked in 

accordance with several such codes. Some of these codes arc public- permitting their reading by 

conventional machines. Others are private, and arc reserved for use by goverimient agencies and the 

like. (C.f. public and private codes in the present assignee's issued patents.) 

Banknotes presently include certain markiiigs which can be used as an aid in note 

authentication. Well known visible structures are added to banknotes to facilitate visiijal authentication 

and machine detection. An example is the seal of the issuing central bank. Others are geometrical 

markings. Desirably, a note is examined by an integrated detection system, for both such visible 

stmctures as well as the present watermark-embedded data, to determine authenticity. 

The visible stiuctures can be sensed using known pattern recognition techniques. Examples of 

such techniques are disclosed in U.S. Pat. Nos. 5,321,773, 5,390,259, 5,533,144, 5,539^41, 5,583,614, 

5,633,952, 4.723,149 and 5,424,807 and laid open foreign application EP 766,449. The embedded 

watermark data can be recovered using the scanning/analysis techniques disclosed in the cited patents 

and publications. 

To reduce counterfeiting, it is desirable that doctiment reproducing technologies recognize 
banknotes and refuse to reproduce same. A photocopier, for example, can sense the presence of either a 
visible structure "or* embedded banknote watermark data, and disable copying if either is present. 
Scanners and printers can be equipped with a similar capability- analyzing the data scanned or to be 
printed for either of these barUcnote hallmarks. If either is detected, the software (or hardware) disables 
further operation. 

The watermark detection criteria provides an important advantage not otherwise available. An 
original bill can be doctored (e.g. by white-out, scissors, or less crude techniques) to remove/obliterate 
the visible structures. Such a document can then be freely copied on either a visible structure sensing 
photocopier or scaimer/printer installation. The removed visible structure can then be added in via a 
second printing/photocopying operation. If the printer is not equipped with banknote-disabling 
capabilities, image-editing tools can be used to insert visible structures back into image data sets scanned 
from such doctored bills, and the complete bill freely printed. By additionally including embedded 
watermark data in the banknote, and sensing same, such ruses will not succeed. 
(A similar ruse is to scan a banknote image on a non-banknote-sensing scanner. The resulting 
image set can then be edited by conventional image editing tools to remove/obliterate the visible 
stmctures. Such a data set can then be printed- even on a printer/photocopier that examines such data 
for the presence of visible structures. Again, the missing visible structures can be inserted by a 
subsequent printing/photocopying operation.) 

Desirably, the visible structure detector and the watermark detector are integrated together as a 

single hardware and/or software tool. Tliis arrangement provides various economics, c.g., in interfacing 

with the scarmer, manipulating pixel data sets for pattern recognition and watermark extraction, 

electronically re-registering the image to facilitate pattern recognition/watermark extraction, issuing 

control signals (e.g. disabling) signals to the photocopier/scanner, etc. 

A related principle is to insert an imperceptible watermark having a UID into all documents 

printed with a printer, scanned with a scanner, or reproduced by a photocopier. The UID is associated 

with the particular printer/photo copier/scanner in a registry database maintained by the products' 

manufecturers. The manufacturer can also enter in this database the name of the distributor to whom the 

product was initial ty shipped. Still further, the owner's name and address can be added to the database 

when the machine is registered for warranty service. While not preventing use of such machines in 

counterfeiting, the embedded UID facilitates identifying the machine that generated a counterfeit 

banknote. (This is an application in which a private watermark might best be used.) 

While the foregoing applications disabled potential counterfeiting operations upon the detection 

of "either* a visible structure or watermarked data, in other applications, both criteria must be met 
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before a banknote is recognized as genuine. Such applications typically involve the receipt or acceptance 
of banknotes, e.g. by ATMs as discussed above. 

The foregoing princples (employing just watermark data, or in conjunction with visible indicia) 

can likewise be used to prevent counterfeiting of tags and labels (e.g. the fiikc labels and tagp commonly 

used in pirating Levis brand jeans, Microsoft software, etc.) 

The reader may first assume that banknote watermarking is effected by slight alterations to the 
ink color/density/distribution, etc. on the paper. This is one approach. Another is to watermark the 
underlying medium (whether paper polymer, etc.) with a watermark. Hiis can be done by changing the 
micro topology of the medium (a la mini- Braille) to manifest the watermark data. Another option is to 
employ a laminate on or within the banknote, where the laminate has the watermarking manifested 
thereon/therein. The laminate can be textured (as above), or its optical transmissivity can vary in 
accordance with a noise-like pattern that is the watermark, or a chemical property can similarly vary. 
Another option is to print at least part of a watermark using photo luminescent ink Tliis allows, 
e.g., a merchant presented with a banknote, to quickly verify the presence of *somc* watermark-like 
indicia in/on the bill even without resort to a scanner and computer analysis (e.g, by examining under a 
black light). Such photolumincscent ink can also print human-readable indicia on the bill, such as the 
denomination of a banknote. (Since irJc-jct printers and other common mass-printing technologies employ 
cyan/magenta/yellow/black to form colore, they can produce only a limited spectrum of colors. 
Photo luminescent colors are outside their capabilities. Fluorescent colors- such as the yellow, pink and 
green dyes used in highlighting markers- can similarly be used and have the advantage of being visible 
without a black light.) 

An improvement to existing encoding techniques is to add an iterative assessment of the 
robustness of the mark, with a corresponding adjustment in a rcwatcrmarking operation. Especially 
when encoding multiple bit watermarks, the characteristics of the underlying content may result in some 
bits being more robusly (e.g. strongly) encoded than others. In an illustrative technique employing this 
improvement, a watermark is first embedded in an objecL Next, a trial decoding operation is performed. 
A confidence measure (e.g. signal -to- noise ratio) associated with each bit detected in the decoding 
operation is then assessed. The bits that appear weakly encoded are identified, and corresponding 
changes are made to the watermarking parameters to bring up the relative strengths of these bits. The 
object is then waterniarked anew, with the changed parameters. This process can be repeated, as needed, 
until all of the bits comprising the encoded data are approximately equally detectable from the encoded 
objea, or meet some predetermined signal-to-noise ratio threshold. 
The foregoing applications, and others, can generally benefit by multiple watermarks. For 
example, an object (physical or data) can be marked once in the spatial domain, and a second time in the 
spatial frequency domain. (It should be understood that any change in one domain has repercussions in 
the other. Here we reference the domain in which the change is directly effected.) 
Another option is to mark an object with watermarks of two different levels of robustness, or 
strength. The more robust watermark withstands various types of corruption, and is detectable in the 
objea even after multiple generations of intervening distortion. The less robust watermark can be made 
frail enough to fail with the first distortion of the object. In a banknote, for example, the less robust 
watermark serves as an authentication mark. Any scanning and reprinting operation will cause it to 
become unreadable. Both the robust and the fraQ watermarks should be present in an authentic 
banknote; only the former watermark will be present in a counterfeiL 

Still another form of multiple watermarking is with content Uiat is compressed. The content can 

be watermarked once (or more) in an uncompressed state. Then, after compression, a further watermark 

(or watermarks) caa be applied. 

Still another advantage from multiple watermarks is protection against sleuthing. If one of the 
watermarks is found and cracked, the other watermark(s) will stUl be present and serve to identify the 
object. 

The foregoing discussion has addressed various technological fixes to many different problems. 
Exemplary solutions have been detailed above. Others will be apparent to the artisan by applying 
common knowledge to extrapolate from the solutions provided above. 
For example, the technology and solutions disclosed herein have made use of elements and 
techniques known from the cited references. Other elements and techniques from the cited references can 
similarly be combined to yield further implementations within the scope of the present invention. Thus, for 
example, holograms with watermark data can be employed in banknotes, single-bit watermarking can 
commonly be substituted for multi-bit watermarking, technology described as using imperceptible 
watermarks can alternatively be practiced using visible watermarks (glyphs, etc.), techniques described 
as applied to images can likewise be applied to video and audio, local scaling of watermark energy can 
be provided to enhance watermark signal-to-noise ratio without increasing human perceptibility, various 
filtering operations can be employed to serve the functions explained in the prior art, watermarks can 
include subliminal graticules to aid in image reregistration, encoding may proceed at the granularity of 
a single pixel (or DCT coe£Bcient), or may similarly treat adjoining groups of pixels (or DCT 
coefficients), the encoding can be optimized to withstand expected forms of content corruption. Etc., etc., 
etc. Thus, the exemplary embodiments are only selected samples of the solutions available by combining 
the teachings referenced above. The other solutions necessarily are not exhaustively described herein, but 
are fairly within the understanding of an artisan given the foregoing disclosure and femiliarity with the 
dted art. 
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I claim: 

1. A method of deterring reproduction of a security 
document, comprising: 

encoding the security document with plural-bit digital 
data, said encoding not being apparent to a human 5 
observer of the security document; 

providing sampled image data corresponding to said 
security document as scanned by a visible light scan- 
ning device; 

analyzing said image data to detect the digital data 
therein, said detection signifying that the image data 
likely corresponds to a security document; and 

intervening in response to detection of the digital data. 

2. The method of claim 1 which includes encoding the 
security document with said digital data across its surface, ^5 
wherein the method cannot be defeated by obliterating 
locali2ed markings characteristic of a security document and 
thereafter reinserting such markings. 

3. The method of claim 1 which further includes: 
analyzing said image data to detect a visible structure 

characteristic of a security document; and 
intervening in response to detection of the visible struc- 
ture; 

wherein detection of either the visible structure or the 25 
digital data serves to trigger the intervention. 

4. The method of claim 1 in which the intervening 
comprises preventing reproduction of the image data. 

5. The method of claim 1 in which the intervening 
comprises sending a message to a remote location to signal 30 
processing of security document data. 

6. The method of claim 1 in which the intervening 
comprises inserting forensic tracing data in the image data. 

7. The method of claim 6 in which the intervening 
comprises inserting first and second plural-bit digital water- 35 
marks in the image data, one of said watermarks serving to 
convey data identifying an apparatus involved in attempted 
reproduction of the security document. 

8. The method of claim 7 in which the first and second 
digital watermarks are of different robustness. 40 

9. The method of claim 7 in which the first and second 
watermarks are encoded in accordance with different code 
signals. 

10. The method of claim 1 in which said encoding 
encompasses regions of the document distinct from any text 45 
or visible security pattern thereon. 

11. A method of deterring reproduction of a security 
document, comprising: 

at a first site, examining suspect image data corresponding 
to an object as sampled by a visible light scanning 50 
device; 

determining at said first site whether said suspect image 
data corresponds to a security document by reference to 
plural-bit digital data stegaaographically encoded in 
the security document, said steganographic encoding ^5 
not being apparent to a human observer of the security 
document; and 

if so, contacting a second site and reporting detection of 
security document-related data. 

12. The method of claim 11 in which said encoding 
encompasses image data corresponding to regions of a 
suspect document distinct from any text or visible security 
pattern thereon. 



13. An apparatus for validating a security document 
comprising: 

a visible light optical scanner for producing image data 
corresponding to a note input to the apparatus; 

a processor for detecting a digital watermaric within the 
image data, the watermark representing plural digital 
bits of auxiliary information, the detection of such 
watermark indicating that the note is a security docu- 
ment; and 

a control unit that responds in a first manner if the 
processor detects a watermark within the image data, 
and that responds in a second, different, manner if the 
processor does not detect a watermark within the image 
data. 

14. The apparatus of claim 13 in which the processor has 
an input through which is a code signal is applied, the code 
signal being required for discerning the watermark from the 
image data. 

15. The apparatus of claim 13 in which: 

the processor also analyzes the unage data for a visible 
structure characteristic of a security document; and 

the control unit responds in the first manner only if both 
the watermark and the visible structure are detected in 
the image data by the processor. 

16. An automatic teller machine according to claim 15, 
said machine serving both to dispense and accept security 
documents from users thereof. 

17. The apparatus of claim 13 in which said watermark 
encompasses image data corresponding to regions of said 
note apart from any text or security pattern represented 
therein. 

18. Apparatus for delecting the attempted counterfeiting 
of security documents, comprising: 

a security document having line art thereon; 
a visible light scanner for scanning the security document; 
and 

a steganographic watermark detector for decoding binary 
data encoded within the security document line art and 
scanned by the scanner, the encoding of said line art 
with binary data not being apparent to a human 
observer of the security document, the presence of said 
decoded binary data indicating the attempted counter- 
feiting of a security document. 

19. Apparatus according to claim 18, further comprising 
a control unit responsive to the decoded binary data for 
initiating insertion of tracer data in an output produced by 
the apparatus. 

20. Apparatus according to claim 19 in which the tracer 
data includes data related to the date of copying. 

21. Apparatus according to claim 19 in which the tracer 
data comprises a steganographic watermark. 

22. Apparatus according to claim 19 in which there is a 
delay in said decoding so that, when scanning a security 
document, the apparatus initially produces output not having 
the tracer data inserted therein, and said output thereafter 
includes the tracer data. 

23. The apparatus of claim 18 in which said encoding 
encompasses regions of the security document distinct from 
any text or visible security pattern thereon. 
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